The creation and storage of digitized data has proliferated in recent years. Accordingly, various storage systems that facilitate efficient and cost effective storage of large amounts of digital data are common today. For example, a storage system may utilize one or more arrays of storage devices, such as in a redundant array of independent disks (RAID), to facilitate the creation, storage, retrieval, and/or processing of digital data. Such a storage system may be implemented using a variety of storage architectures, such as a network-attached storage (NAS) environment, a storage area network (SAN), a direct-attached storage environment, etc., and a variety of levels, such as RAID 0, RAID 1, RAID 2, RAID 3, RAID 4, RAID 5, etc, and combinations thereof. The storage devices utilized by the storage system may comprise various configurations of storage devices, such as hard disk drives, solid state drives, flash memory, etc.
The storage system may include interfaces for one or more application system (e.g., client system, server system, etc.) to interact with the storage system for storing and accessing digital data within storage devices of the storage system. A storage system controller may be provided as part of the storage system, disposed between the application system and the storage devices of the storage arrays, to facilitate communication of data therebetween. For example, a storage system controller may be implemented upon a host processor-based system of the storage system which is configured to include appropriate hardware (e.g., input/output, memory, processor, etc.) and logic (e.g., instruction sets, software, firmware, etc.) operable to facilitate data storage using one or more associated storage device arrays. A common architecture for a storage system storage device controller includes a controller cache, typically implemented using dynamic random access memory (DRAM), to hold data being read from and/or written to the storage devices of the storage system. Such a cache is typically organized into cache blocks which are used to hold the cached data.
For storage devices implementing small computer system interface (SCSI) logical units (LU) with a write through (WT) cache mode, the data for writing to the storage devices is inserted into the controller cache and then immediately written out from this cache to the appropriate storage devices. A “good” status indicator is not returned to the host until the data has been written successfully to the storage devices.
However, for storage devices implementing SCSI logical units (LU) with a write back (WB) cache mode, the data for writing to the storage devices is inserted into the controller cache, the status is returned to the host when the data is inserted into the cache, and at some later point in time the data is written from the cache to the appropriate storage devices. During the period of time between the insertion of the data into the cache and writing the data to the storage devices, the data in the cache is inconsistent with the corresponding data on the storage devices. The data in the cache is thus referred to as dirty data. Once the dirty cache blocks have been successfully written to the appropriate storage devices (referred to as flushing the dirty data), the cache blocks are marked as clean.
In both the WT and WB cache mode cases, write latency associated with writing data from the cache to the corresponding storage devices can increase significantly due to various phenomena. For example, one or more storage devices may misbehave (i.e., experience a period of aberrant operation, operate outside of performance specifications for a period of time, fail, etc.) resulting in delayed and/or failed data write operations. Such misbehavior may, for example, be associated with a storage device internal condition (e.g., an error recovery/retry algorithm), and is often not associated with a storage device malfunction per se. In particular, many storage devices (e.g., relatively inexpensive hard disk drives) provide a wide performance parameter range which is considered “normal” or “acceptable”, and thus may present significant write latency variation in their normal operation. Excessive data write latency, whether remaining within the normal storage device operating parameters (e.g., published operating specifications), are nevertheless referred to herein as storage device misbehavior. Irrespective of the particular source of the storage device misbehavior, such storage device misbehavior may result in undesired storage system operation.
Some applications providing storage of digitized data, such as streaming media applications (e.g., high definition video surveillance), require that write latency associated with writing data to the underlying storage devices is maintained within specific, relatively narrow limits in order to avoid data loss and/or other undesirable results. In particular, if the write latency of one or more storage device increases significantly for some period of time, the delay in writing data and thus cleaning the storage system data cache can cause undesired results in other system components. For example, such increased write latency may result in an application system (e.g., media server) which is accessing the storage system to run out of buffer space and start dropping data (e.g., frames of the streaming video provided by the aforementioned high definition surveillance system may be dropped, thereby resulting in poor quality video images).
In the WT cache mode case, the foregoing data write latency increases will directly increase the latency of the actual host write since the host cannot complete until all the underlying data writes are complete. In the WB cache mode case, the foregoing data write latency increases will delay marking the dirty cache blocks clean so that these cache blocks can be used for other writes (or reads) by the storage device controller.
In a storage system having a correctly sized storage device cache, the storage device controller cache flush process will be able to keep up with the host write requests so there will always be clean or free cache blocks available for new host writes. However, if the flush process cannot keep up with the host write arrival rate, such as due to data write latency associated with a misbehaving storage device, the storage device controller cache will eventually become saturated and a host write request will have to wait until cache blocks become free/clean.
Potentially increasing the opportunity for a storage device cache, or some portion thereof, to become saturated, some storage arrays establish a specific limit on how many dirty cache blocks a particular drive group may have. This means that a host data write for a LU in a drive group may be required to wait for clean/free cache blocks when the dirty cache block limit for the drive group has been reached, even though the storage device cache may not have been fully saturated.
In operation, a storage device controller may write data to the storage devices in stripes (i.e., a stripe comprises a block of data written across a plurality of storage devices using chunks, wherein a chunk is the unit of data to be written to a single storage device). Accordingly, the storage device controller may not mark the storage device cache blocks of a stripe clean until all the cache blocks in the stripe have been written to the storage devices. This means that a single misbehaving storage device in a storage device group can delay cleaning not only the cache blocks corresponding to the misbehaving storage device but all cache blocks corresponding to the storage devices of the entire storage device group for the stripe being written.
From the foregoing, it can readily be appreciated that data write latency associated with writing data to one or more storage devices of a storage system can relatively quickly result in appreciable performance degradation and/or unsatisfactory operation of the storage system. Even where a caching mode, such as the aforementioned WB caching mode, is enabled, a misbehaving storage device or other system issue in the storage system can cause data writes to be delayed and/or data to be lost as the storage device cache limit is reached.
Although it may be possible to enlarge the storage device cache in an attempt to accommodate temporary storage device misbehavior, such a solution has been found to be unsatisfactory. In particular, such storage system host level storage device caches are comprised of DRAM, which is a relatively expensive resource. The amount of storage device cache memory may be on the order of a plurality of gigabytes for each volume or other container for which storage device caching is to be provided. The cost to implement such added memory to a large storage system configuration has typically proven undesirably high.
Another potential way to address the foregoing storage device misbehavior is to utilize higher quality storage devices. For example, rather than populating a storage device array with a plurality of relatively inexpensive hard disk drives, higher quality hard disk drives providing a more narrow range of normal or acceptable operating parameters may be used. Such a solution, however, can be relatively expensive due to the fact that not only are the individual hard disk drives more expensive, but due to the number of hard disk drives used. Moreover, such a solution is contrary to a fundamental tenet of RAID (i.e., inexpensive).